Skip to content

UI: Fix slow pending Dag run checks on large databases - #75674

Merged
bbovenzi merged 1 commit into
apache:mainfrom
yuseok89:speed-up-pending-runs-check
Oct 27, 2026
Merged

bbovenzi merged 1 commit into
apache:mainfrom
yuseok89:speed-up-pending-runs-check

Conversation

@yuseok89

@yuseok89 yuseok89 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

The dashboard polls for running or queued Dag runs to decide whether to auto-refresh, every 3 seconds while runs are active. The check fetched one Dag run ordered by id. When the statistics from the last ANALYZE count some pending runs, Postgres serves that by scanning dag_run by primary key from the oldest run, and since pending runs are the newest it reads nearly the whole table. Only the count is used, so the check now asks for no rows and reads total_entries.

Measured on Postgres with 5M finished runs, through the real endpoint.

Running or queued runs Before (limit=1) After (limit=0)
70 3.3 to 11.8 s 1 ms

The new test serves the endpoint from an msw handler that returns only limit rows with the full count in total_entries, as the API does. It checks that refresh still turns on and that every request asks for limit=0, and it fails on main.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    • Claude Code (Opus 5.5)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added the area:UI Related to UI/UX. For Frontend Developers. label Oct 26, 2026
@yuseok89
yuseok89 marked this pull request as ready for review October 26, 2026 23:44
@bbovenzi bbovenzi added the backport-to-v3-4-test Backport to v3-4-test label Oct 27, 2026
@bbovenzi bbovenzi added this to the Airflow 3.4.0 milestone Oct 27, 2026
@bbovenzi
bbovenzi merged commit 91ca0ac into apache:main Oct 27, 2026
155 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-4-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
✅ v3-4-test PR Link

bbovenzi pushed a commit that referenced this pull request Oct 27, 2026
…75674) (#75799)

(cherry picked from commit 91ca0ac)

Co-authored-by: Yuseok Jo <yuseok89@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:UI Related to UI/UX. For Frontend Developers. backport-to-v3-4-test Backport to v3-4-test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants